Guidelines: The do's, don'ts and don't knows of direct observation of clinical skills in medical education.

IntroductionDirect observation of clinical skills is a key assessment strategy in competency-based medical education. The guidelines presented in this paper synthesize the literature on direct observation of clinical skills. The goal is to provide a practical list of Do's, Don'ts and Don't Knows about direct observation for supervisors who teach learners in the clinical setting and for educational leaders who are responsible for clinical training programs.MethodsWe built consensus through an iterative approach in which each author, based on their medical education and research knowledge and expertise, independently developed a list of Do's, Don'ts, and Don't Knows about direct observation of clinical skills. Lists were compiled, discussed and revised. We then sought and compiled evidence to support each guideline and determine the strength of each guideline.ResultsA final set of 33 Do's, Don'ts and Don't Knows is presented along with a summary of evidence for each guideline. Guidelines focus on two groups: individual supervisors and the educational leaders responsible for clinical training programs. Guidelines address recommendations for how to focus direct observation, select an assessment tool, promote high quality assessments, conduct rater training, and create a learning culture conducive to direct observation.ConclusionsHigh frequency, high quality direct observation of clinical skills can be challenging. These guidelines offer important evidence-based Do's and Don'ts that can help improve the frequency and quality of direct observation. Improving direct observation requires focus not just on individual supervisors and their learners, but also on the organizations and cultures in which they work and train. Additional research to address the Don't Knows can help educators realize the full potential of direct observation in competency-based education

Factors influencing responsiveness to feedback: on the interplay between fear, confidence, and reasoning processes

Self-appraisal has repeatedly been shown to be inadequate as a mechanism for performance improvement. This has placed greater emphasis on understanding the processes through which self-perception and external feedback interact to influence professional development. As feedback is inevitably interpreted through the lens of one’s self-perceptions it is important to understand how learners interpret, accept, and use feedback (or not) and the factors that influence those interpretations. 134 participants from 8 health professional training/continuing competence programs were recruited to participate in focus groups. Analyses were designed to (a) elicit understandings of the processes used by learners and physicians to interpret, accept and use (or not) data to inform their perceptions of their clinical performance, and (b) further understand the factors (internal and external) believed to influence interpretation of feedback. Multiple influences appear to impact upon the interpretation and uptake of feedback. These include confidence, experience, and fear of not appearing knowledgeable. Importantly, however, each could have a paradoxical effect of both increasing and decreasing receptivity. Less prevalent but nonetheless important themes suggested mechanisms through which cognitive reasoning processes might impede growth from formative feedback. Many studies have examined the effectiveness of feedback through variable interventions focused on feedback delivery. This study suggests that it is equally important to consider feedback from the perspective of how it is received. The interplay observed between fear, confidence, and reasoning processes reinforces the notion that there is no simple recipe for the delivery of effective feedback. These factors should be taken into account when trying to understand (a) why self-appraisal can be flawed, (b) why appropriate external feedback is vital (yet can be ineffective), and (c) why we may need to disentangle the goals of performance improvement from the goals of improving self-assessment

Longitudinal Milestone Assessment Extending Through Subspecialty Training: The Relationship Between ACGME Internal Medicine Residency Milestones and Subsequent Pulmonary and Critical Care Fellowship Milestones

Purpose Accreditation Council for Graduate Medical Education (ACGME) milestones were implemented across medical subspecialties in 2015. Although milestones were proposed as a longitudinal assessment tool potentially providing opportunities for early implementation of individualized fellowship learning plans, the association of subspecialty fellowship ratings with prior residency ratings remains unclear. This study aimed to assess the relationship between internal medicine (IM) residency milestones and pulmonary and critical care medicine (PCCM) fellowship milestones. Method A multicenter retrospective cohort analysis was conducted for all PCCM trainees in ACGME-accredited PCCM fellowship programs, 2017–2018, who had complete prior IM milestone ratings from 2014 to 2017. Only professionalism and interpersonal and communication skills (ICS) were included based on shared anchors between IM and PCCM milestones. Using a generalized estimating equations model, the association of PCCM milestones ≤ 2.5 during the first fellowship year with corresponding IM subcompetencies was assessed at each time point, nested by program. Statistical significance was determined using logistic regression. Results The study included 354 unique PCCM fellows. For ICS and professionalism subcompetencies, fellows with higher IM ratings were less likely to obtain PCCM ratings ≤ 2.5 during the first fellowship year. Each ICS subcompetency was significantly associated with future lapses in fellowship (ICS01: β = −0.67, P = .003; ICS02: β = −0.70, P = .001; ICS03: β = −0.60, P = .004) at various residency time points. Similar associations were noted for PROF03 (β = −0.57, P = .007). Conclusions Findings demonstrated an association between IM milestone ratings and low milestone ratings during PCCM fellowship. IM trainees with low ratings in several professionalism and ICS subcompetencies were more likely to be rated ≤ 2.5 during the first PCCM fellowship year. This highlights a potential use of longitudinal milestones to target educational gaps at the beginning of PCCM fellowship

Racial and ethnic differences in internal medicine residency assessments

IMPORTANCE: Previous studies have demonstrated racial and ethnic inequities in medical student assessments, awards, and faculty promotions at academic medical centers. Few data exist about similar racial and ethnic disparities at the level of graduate medical education. OBJECTIVE: To examine the association between race and ethnicity and performance assessments among a national cohort of internal medicine residents. DESIGN, SETTING, AND PARTICIPANTS: This retrospective cohort study evaluated assessments of performance for 9026 internal medicine residents from the graduating classes of 2016 and 2017 at Accreditation Council of Graduate Medical Education (ACGME)-accredited internal medicine residency programs in the US. Analyses were conducted between July 1, 2020, and June 31, 2022. MAIN OUTCOMES AND MEASURES: The primary outcome was midyear and year-end total ACGME Milestone scores for underrepresented in medicine (URiM [Hispanic only; non-Hispanic American Indian, Alaska Native, or Native Hawaiian/Pacific Islander only; or non-Hispanic Black/African American]) and Asian residents compared with White residents as determined by their Clinical Competency Committees and residency program directors. Differences in scores between Asian and URiM residents compared with White residents were also compared for each of the 6 competency domains as supportive outcomes. RESULTS: The study cohort included 9026 residents from 305 internal medicine residency programs. Of these residents, 3994 (44.2%) were female, 3258 (36.1%) were Asian, 1216 (13.5%) were URiM, and 4552 (50.4%) were White. In the fully adjusted model, no difference was found in the initial midyear total Milestone scores between URiM and White residents, but there was a difference between Asian and White residents, which favored White residents (mean [SD] difference in scores for Asian residents: -1.27 [0.38]; P \u3c .001). In the second year of training, White residents received increasingly higher scores relative to URiM and Asian residents. These racial disparities peaked in postgraduate year (PGY) 2 (mean [SD] difference in scores for URiM residents, -2.54 [0.38]; P \u3c .001; mean [SD] difference in scores for Asian residents, -1.9 [0.27]; P \u3c .001). By the final year 3 assessment, the gap between White and Asian and URiM residents\u27 scores narrowed, and no racial or ethnic differences were found. Trends in racial and ethnic differences among the 6 competency domains mirrored total Milestone scores, with differences peaking in PGY2 and then decreasing in PGY3 such that parity in assessment was reached in all competency domains by the end of training. CONCLUSIONS AND RELEVANCE: In this cohort study, URiM and Asian internal medicine residents received lower ratings on performance assessments than their White peers during the first and second years of training, which may reflect racial bias in assessment. This disparity in assessment may limit opportunities for physicians from minoritized racial and ethnic groups and hinder physician workforce diversity